Skip to content

feat(rollout): add ordered trajectory reads - #158

Merged
beinan merged 1 commit into
lance-format:mainfrom
xuzha:feat/trajectory-read
Jul 21, 2026
Merged

feat(rollout): add ordered trajectory reads#158
beinan merged 1 commit into
lance-format:mainfrom
xuzha:feat/trajectory-read

Conversation

@xuzha

@xuzha xuzha commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add get_trajectory(rollout_id) to the shared rollout API, returning rows ordered by (sequence_order, id)
  • reuse the existing filtered read path so local reads span base fragments and MemWAL shards, while remote reads require no new endpoint
  • expose ordered trajectory reads through PyO3 and the synchronous/asynchronous Python wrappers
  • reject empty rollout IDs and cover local multi-fragment plus remote behavior

Testing

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace
  • uv run pytest -q python/tests/test_rollout.py python/tests/test_rollout_remote.py (13 passed)
  • uv run ruff format --check python/
  • uv run ruff check python/
  • uv run pyright

Notes

  • The repository CI helper still references the removed rust/lance-context/Cargo.toml path, so equivalent checks were run against the current root workspace.
  • A broad local uv run pytest completed with 186 passed, 2 skipped, 1 xfailed, and 10 unrelated failures in existing Moto/S3 setup and DummyInner context search/list test signatures. The changed rollout suites pass in full.

@beinan
beinan merged commit 722a5ba into lance-format:main Jul 21, 2026
9 checks passed
beinan added a commit that referenced this pull request Jul 25, 2026
## The problem

`python-test.yml` ran `pytest python/tests/` with `working-directory:
python`, which resolves to **`python/python/tests/`** — a single stale
file with 3 tests. The real suite, **`python/tests/` (24 files, 179
tests)**, was never collected. pytest exits 0 on a
valid-but-nearly-empty directory, so this failed silently.

Git history dates the split: `python/python/tests/` last changed in #59;
`python/tests/` is current through #158. **~100 PRs merged ungated** —
rollout store, async wrappers, export, ingestion.

Two further layers of silencing, each sufficient on its own:
- Only bare `pytest` was installed, never the `[tests]` extra — so
`pytest-asyncio` was absent.
- No `asyncio_mode` anywhere, so the 12 `async def` tests were
collected, never awaited, and **passed without executing a line**.

The same path bug appeared three more times: the doctest step was a
permanent no-op (an `if` on a path that never exists, plus `|| echo`),
and `ruff format --check python/` / `ruff check python/` **never linted
the 24 test files**. `.codex/.../run_ci_checks.sh` pointed at a `rust/`
directory that does not exist, so `set -e` killed it before any Python
check ran.

## CI changes

- Pin collection with `testpaths = ["tests"]` so the ambiguous relative
path cannot resurface.
- `asyncio_mode = "strict"` + `--strict-markers --strict-config` — a
mistyped marker now fails instead of silently disarming a test.
- Install `[lance-python,tests]`; lint `.` instead of `python/`; make
the doctest step real.
- Point the helper script at the real `crates/` layout.

## Turning the suite on surfaced 19 real failures

**1. Embedded `RolloutStore` silently discarded every write.** This is
the significant one. #181 made `add` durable-but-not-visible, with
visibility driven by a server-side sweeper. The embedded path has no
sweeper, and `flush()` was never exposed to Python — so:

```python
s = RolloutStore.open(d)
s.add({'id':'a','rollout_id':'t1'})   # -> {'version': 2, 'ids': ['a'], 'count': 1}
s.list()                              # -> []   ... and still [] after reopen
```

Data acknowledged, then unreadable forever. This PR exposes `flush()`
through the facade, PyO3, and both sync and async Python wrappers.

**2. `test_search.py`'s `DummyInner` fake had drifted** from the real
binding signature (missing `include_binary`/`include_embedding`), so all
55 tests raised `TypeError` on contact with current code.

**3. Remote rollout tests** asserted read-your-write against a server
whose default flush interval is 30s. They now set
`ROLLOUT_FLUSH_INTERVAL_SECS=1` and poll — asserting the row *arrives*
rather than pinning to the interval.

The 3 stale tests are **moved** into `python/tests/` rather than dropped
(they cover `snapshot`/`fork`/`memory://` and still pass), and the
orphaned directory is removed so nothing can silently collect it again.

## Verification

- **182 passed, 2 skipped** (179 real + 3 rescued).
- `ruff format --check .`, `ruff check .`, `pyright` — all clean.
- `cargo fmt --all -- --check` and clippy clean on the changed crates.
- Confirmed the async tests now genuinely execute by injecting a failing
assertion and watching it fail — it would not have before this change.

## Note on scope

This is deliberately scoped to *making CI honest* plus the failures that
gating surfaced. It does **not** address: publish workflows having no
`needs:` on any test job (a release cut from a red `main` ships to
PyPI/crates.io), `cargo test` covering only 2 of 8 crates and `--lib`
only, or the `DummyInner` fake asserting hardcoded literals so RRF
ranking is still never exercised. Happy to file those separately.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants